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Abstract. Disordered and frustrated graphical systems are ubiquitous in physics, 
biology, and information science. For models on complete graphs or random graphs, 
deep understanding has been achieved through the mean-field replica and cavity 
methods. But finite-dimensional 'real' systems persist to be very challenging because 
of the abundance of short loops and strong local correlations. A statistical mechanics 
theory is constructed in this paper for finite-dimensional models based on the 
mathematical framework of partition function expansion and the concept of region- 
graphs. Rigorous expressions for the free energy and grand free energy arc derived. 
Message-passing equations on the region-graph, such as belief-propagation and survey- 
propagation, are also derived rigorously. 
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Graphical models are used to describe systems composed of elements which lack 
translational degrees of freedom but have changeable internal states. Disordered and 
frustrated graphical models are ubiquitous in physics (spin glasses), biology (gene 
regulation and neuron networks), and information science (error-correcting codes, 
constraint satisfaction problems). Revealing and understanding the rich and fascinating 
properties of these systems has been a major field of statistical physics for many years 
[H 12]. On the theoretical side, a deep understanding of the equilibrium behaviors of 
models defined on complete graphs (e.g., the Sherrington-Kirkpatrick model) or finite- 
connectivity random graphs (e.g., the Viana-Bray model) has been achieved by the 
mean- field replica and cavity methods [21 SI El El E] • 

Due to the finite-dimensional nature, real- world graphical model systems (e.g., 
the Edwards- Anderson (EA) model [H]) usually contain extremely many short loops 
with strong interaction strengths, which cause strong local correlations within groups of 
elements. The abundance of short loops makes theoretical study of finite-dimensional 
graphical models very difficult, and so far progress has mainly been achieved through 
extensive numerical simulations (see, e.g., [9] and references therein). The high- 
temperature equilibrium properties of finite-dimensional models can be studied by the 
cluster variation method fiU\ ITT] and its recent extension, the region-graph method [12] . 
But for the most interesting low-temperature regime, a powerful and general theoretical 
framework is still lacking. Recently, Rizzo and co-workers [13j made an initial step 
in combining the cluster variation method with the replica method to study the low- 
temperature behavior of the EA model, but several technical difficulties of this approach 
still need to be remedied (such as the issue of non-positive-definite message functionals) . 

In this letter, a statistical mechanics theory for finite-dimensional graphical models 
is rigorously constructed based on the mathematical framework of partition function 
expansion [TH [T3] and the region-graph concept of Yedidia and co-workers [12]. In 
this theoretical framework, the sets of elements that form local loops are regarded 
as regions, and the possible strong local correlations within the regions are explicitly 
taken into account. Free energy expressions and region- graph message-passing equations 
at different hierarchies of the free energy landscape are derived without using any 
approximations. The theoretical framework directly corresponds to a distributed 
algorithm for single graphical systems. This work may find broad applications in finite- 
dimensional spin-glass systems and in real-world information systems (such as image 
processing). 

Region-graph representation — We consider a general model system defined on a 
factor-graph [TB] of variable nodes {i = 1,2, . . . , N, representing the elements) and 
M function nodes (a = 1,2, ...,M, representing the interactions). The total-energy 
function has the following additive form: 



A configuration x of the system is defined by the states of all the variable nodes, 
X = {xi,X2, . . . ,xn}, Xi being the discrete state of node i. Each function node a 



M 




(1) 



a=l 
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Figure 1. Part of the factor-graph for the two-dimensional Edwards- Anderson model 
on a square lattice. Circles represents variable nodes and squares represent interactions. 



is connected to a set da of variable nodes, its energy Ea depends only on the states 
= {xi\i E da} of the variable nodes in da. As a simple example, figure [l] shows the 
factor-graph for the EA model on a two-dimensional (2D) square lattice, whose energy 
function is defined as 

H{x) = J2 JijXiXj, (2) 

where (z, j) denotes an edge between two nearest-neighboring lattice sites i and j, Jij is 
the coupling constant on this edge (the state of each variable node is binary in the EA 
model, Xi G {— 1,-|-1}). The equilibrium partition function of the general system Q is 
defined as 

M 

Z{P)^Y.^~P^^''^=Y.\{U^8a\ (3) 

X X a=l 

where /3 is the inverse temperature, and the nonnegative function ipai^da) = e^^^°-^-9a) 
is the Boltzmann factor of function node a. 

The factor-graph of model ([T]) might contain many short loops, which cause strong 
local correlations among the variable nodes. To better describe the correlations and 
fluctuations caused by short loops in the system, it is desirable to view the factor-graph 
as a union of different regions. Each region contains a set of variable nodes and function 
nodes. The variable nodes that appear in the same region are supposed to be strongly 
correlated and are treated with special care. More precisely, a region-graph R is formed 
by regions {a, 7, /x, . . .} and the edges {(a, 7), (/x, i/), . . .} between regions |I2]. A region 
a contains some variable nodes and function nodes of the factor graph, it satisfies the 
condition that, if a function node a G a, then all the variable nodes connected to a are 
in a (that is, da G a). The region concept is an extension of the cluster concept of the 
cluster variation method [TUl E]. The edges in a region-graph are directed. If there is 
an edge pointing from a region fi to another region u, then u must be a subregion of /i 
(namely, if variable node z G z/, then i E fJ^, and if function node a G z/, then a E fi). If 
there is a directed path from region a to region 7, then we say that 7 < a and a > 7 
(notice that if there is no directed path from a to 7, such an ordering relationship does 
not exist, even if 7 C a). 
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Figure 2. Part of a region-graph for tire 2D Edwards- Anderson model, whose 
factor-graph is shown in figure [l] There are two types of regions, the square 
regions (containing four variable nodes and four function nodes) and the pair regions 
(containing two variable nodes and one function node). The red short lines represent 
the edges of the region-graph. Each square region has a counting number c — 1, and 
each pair region has a counting number c = — 1. 



Each region a of the region-graph has a counting number Cq, that is recursively 
determined by 

Ca = l- ^7- (4) 

{7|7G-R,7>a} 

Notice that J^-yya c-y = 1 for any region a E R, that is, each region is counted exactly once 
(see [T2|, [H]). The region-graph R and its associated counting numbers are required to 
satisfy the following region-graph condition [12]: For any variable node i, the subgraph 
of R formed by all the regions a containing i and all the edges between these regions is 
connected; furthermore, 



{a\i£a} 



1. 



(5) 



For the same factor-graph it is usually possible to construct many different region- 
graphs which all satisfy the region-graph condition. The factor-graph itself can be turned 
into the simplest region-graph with two types of regions (each small region is formed by 
a single variable node, and each large region is formed by a function node and all the 
connected variable nodes). Constructing an appropriate region-graph for a graphical 
model ([T]) is by no means a trivial issue, a lot of physical insights are needed here. We 
leave this construction problem as an open issue for future study, and simply assume a 
suitable region-graph R for model ([T]) is ready for use. To give a simple example, we 
show in figure [2] part of a region-graph for the EA model (|2]) on a 2D square lattice. 

Partition function expansion — Because of the region-graph condition ([s]), we can 
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reformulate the partition function ([s]) as 

Z{P) = Y.]l\]lM^aa)r- (6) 

For each region a we define its configuration as = {xf\i G a}, where xf is the state 
of node i in a. For a function node a in region a, the variable states at its neighborhood 
is then denoted as xfg^, with xfg^ = {x'^\i G da}. The Boltzmann factor for region a is 

denoted as ^aix^) = Uaea'^a'^i^da)- 

A variable node i may belong to two or more regions (say a,7, ...). If this is 
the case, we regard the states xfjx], ... of node i in the different regions as different 
variables. With these auxiliary states {xda G R}, we then have 



m = E E n 



7 

Xj , Xi 



(7) 
(8) 



= E n n n ^( 

where the Kronecker symbol 6{x, y) = 1 if x = y and 6{x, y) = otherwise. The 
equivalence of ^ and ([T]) is ensured by the region-graph condition that the subgraph 
induced by any variable node i is connected. In ([s]), the product over all the edges (/i, u) 
of the region-graph R makes sure that, the partition function Z is contributed only by 
those region configurations for which each variable node i takes the same state in all the 
different regions that contain i. These are the edge constraints on R. We notice that, 
if in R there are two or more directed paths from a region a to another region 7, then 
some of the region-graph edge constraints are redundant in the sense that they can be 
removed from the edge product without affecting the equivalence of ([s]) and ([T]). It is 
desirable for a region-graph R to be free of redundant edges. If some edges of R are 
redundant, it should not be included in the edge product of (|8]). In what follows, for 
simplicity, we assume that the region-graph R contains no redundant edges. 

For each edge (/x, u) G i? we introduce two auxiliary probability distributions 
and py-^^{x^f^y). They are at the moment arbitrary functions under the 



Pfj.-¥u\X_ppfp^/ 

minimal constraints of being nonnegative and normalized. The state vector x^^^, is 
defined as x^r\y = {Xi\i G fl u}, similarly x„^^ = G /i fl z/}. We then get the 

following formula of central importance 



E n 

{xJaeR 



n Pi- 



[X 



n 

{^J.,u)eR 



P^^u {x_^nfj, )Pu-^fi {x^ni/) 



(9) 



where da denotes the set of regions that are connected with region a by an edge. A 
loop series expression for the partition function ^ can be easily written down following 
[H]. The equilibrium free energy F{(3) = — (1//3) In of the system is expressed as 



Fo-^ln 



1+ E ^ 



rioo 



(10) 



where rioop denotes any subgraph that is free of dangling edges (each a G rioop is 
connected with at least two other regions of rioop), and L^.^^^ is its correction contribution 
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to the free energy. To ensure that any subgraph r with at least one dangling edge has zero 
correction contribution, the auxiliary probability functions need to satisfy the following 
equation 



X 



111 



This self-consistent equation is referred to as the region-graph belief-propagation (rgBP 
equation. 

The free energy Fq in (|To|) is expressed as 



(12) 



with 



/7 --^In 



7 12^.700 J 



E ^7(3^7) n p 

x^ aed'y 

E Pfj.^u{Xfj.nu)Pu^fi{.3liMn 



The loop correction has the following expression 



^r= E 11^7(317) n 



(13) 



{x^|7er} 7er 



with 



A 



W^{X^} 



>7l2i7naJ 



^7(^7) naga7Po 

^x ^7(217) nae<97 Pa-!>7 (2^-700) 



At a fixed point of the rgBP equation (11), -Fq gives an approximate expression for 
the true free energy of the system, when all the loop corrections in (10) are neglected. 
A nice property of Fq is that, when viewed as a functional of the probability functions 
[p^^y^Py^fj], the first variation of Fq vanishes at a fixed point of (11). In other words, 
Fq attains its extremal value at a rgBP fixed point. Because of this extremal property, 
we regard a fixed point of (11 ) as a macroscopic state of the model ([T]), with extremal 
free energy value Fq. Using the counting number expression (Q, we can re-express Fq 
as 



Fq= E [^7 + E Ca] A - E + E Ca] 



"fGR o>7 



(14) 



where (/i — )■ u) denotes a directed edge of R from region to region i/, and fa is the 
free energy of region a as expressed by 

/«=E/7- E fM- (15) 

7<a {ii,u):ij,<a,u<a 
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Notice (14) has the same formal expression as the region-graph variational free energy 



expression Frg of [12]. But Fq and Fr^ in general may not be equivalent. 

The statistical mechanics description of a single macroscopic state of model ([T]) is 



composed of (10) and (11). It can be regarded as forming a replica-symmetric spin- 



glass theory in the region-graph representation. If the typical length of the loops of the 



region-graph is long enough, since each ^(p,iy) in the product of (13) has zero mean value 
under the probability measures {w^{x^)}, the loop correction contributions to the free 
energy might be negligible within each macroscopic state of the system. 

For the region-graph shown in figure [2] representing the the square-lattice EA model, 
the explicit form of the rgBP equation ( [TT| ) can be easily obtained. We have applied the 
rgBP equation on single instances of the 2D EA model and obtained quite satisfactory 
results (will be reported in a full paper). In the case that all the coupling constants 
Jij = J > (the Ising model), the rgBP equation predicts a phase-transition from 
the paramagnetic phase to the ferromagnetic phase at a critical temperature value 
T = 2.65635J. This value is lower than the value of T = 2.8854J as obtained by 
the Bethe-Peierls approximation, but still higher than the exact critical temperature 
T = 2.2692J. Further improvements on the critical temperature prediction should be 
achievable by working on region- graphs with larger regions. 

Grand partition function expansion — For graphical models with strong disorder 



and frustration, very probably the rgBP equation (11) has multiple fixed points at 



low enough temperatures. This situation corresponds to the existence of multiple low- 
temperature macroscopic states s of the model ([TJ, each of which is characterized by an 
extremal free energy value Fq^\ A full understanding of the equilibrium property of the 
system then requires a quantitative description of the free energy landscape at the level 
of macroscopic states. For this purpose, we follow [H] (see also [H [5]) and introduce a 
grand partition function E{y; (3) 



'Dp^^u J I)p„^f,5{pf,- 



-yFo 



(16) 



= n 

Because of the Dirac delta functions, only the rgBP fixed points contribute to the 
grand partition function. The parameter y is the inverse temperature at the level of 
macroscopic states. For each edge (/i, u) of the region-graph we introduce two auxiliary 
probability functions Pf,^y{p^^y) and Py^^{py 

S(i/;/3) = 



) and rewrite ( 16 ) as 



n n 



Dp 



-yfa 



n 



5{p^-,y - B^^y)6{p„^^ - B, 
e-yh.,^)P^_,{p^^,)P,^^{p 



■ (17) 



We can then follow [13] and obtain a loop series for the expression (17). The grand 
free energy of the system G{y; (3) = {—1/y) lnS(?/; (3) is written as 

In 1 + > W 

oop 



G{y;^) = Go-^\n 



1+ E 4:^ 

r J ' loo 

rioop^R 
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where the leading term Go has the expression 



E^7 

■yeR 



with 



9M 



In 



y 
1 

y 



n 



J2 



(19) 



a—>-'y) 



-yfi 



In 



Dp 



The correction contribution L^^^ of a subgraph to the grand free energy has a similar 
expression as (13) [Tl]. To ensure that any subgraph with at least one dangling edge 
has vanishing correction contribution to the grand free energy, each probability function 
P^^y{p^^y) needs to satisfy the following equation 



where 



n i^pt 



. P .in . )p-yf)^ 



'Hp, 



B 



n 



J Dp. 



'j—^fiP-y- 



Xp 



-yffj. 



(20) 



n Pi- 



Equation (20) is called the region-graph survey-propagation (rgSP) equation. If we 
neglect loop correction contributions, Go then gives an approximate expression for the 
true grand free energy. By knowing the grand free energy values at different inverse 
temperatures y, we can then calculate the number exp[iVS(/)] of macroscopic states 
with given extremal free energy density / = Fq/N [5l[Tl]. S(/) is called the complexity 
function in spin-glass literature It gives a quantitative description of the free energy 
landscape of the Fq functional ( [l2| ). Notice that Gq can again be viewed as a functional 
of the probability distributions {P^^,^, P^^^}. It is easy to check that the first variation 
of Gq vanishes at a fixed point of (20), suggesting that each rgSP fixed point corresponds 
to an extremal value of Gq. 

The statistical mechanics description of the model ([!]) at the level of macroscopic 
states is composed of ( [Tsj ) and ( [20| ). This description can be regarded as the first-step 
replica-symmetry-breaking (IRSB) spin-glass theory in the region-graph representation. 

If the free energy functional Go has multiple extremal values, then a higher-level 
partition function needs to be defined similar to (16), and the free energy landscape of 
Gq will then be described by a higher-level message-passing equation. This hierarchy of 
partition function expansions can be continued to even higher levels if necessary, leading 
to a mathematical theory that can be regarded as the full-step RSB spin-glass theory 
in the region-graph representation. 

Conclusion and discussions — For a general graphical model ([T]) represented through 
a region-graph, a statistical mechanics theory was rigorously derived in this work 
from the mathematical framework of partition function expansion [T3]. The obtained 
free energy expressions and message-passing equations are exact in the sense the no 
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assumptions nor approximations were made during the theoretical construction. This 
theory is most suitable for studying finite-dimensional models which are full of short 
loops and strong local correlations. The theory reduces to that of [H] in the special 
case of the region-graph being a factor-graph. 

Yedidia and co-authors [12] constructed a variational free energy based on some 
heuristic arguments and consistency requirements, and they then obtained several 
sets of generalized belief-propagation equations. The starting point of the variational 
approaches of [12] and [IHl HI] is the non-equilibrium Gibbs free energy functional, while 
our starting point is the equilibrium partition function. Because of this qualitative 
difference, the free energy expression Fq in general may not be equivalent to the 
variational free energy expression of [12]. In an accompanying full paper we will 



investigate this issue more systematically. A comparison between the rgBP (11) and 
the generalized belief-propagation schemes of [12j will also be made. The possible links 
between our approach and the replica cluster variation method of [13] will also be 
thoroughly discussed in the accompanying paper. 

A distributed algorithm of message-passing can be implemented on single finite- 
dimensional graphical models. To be efficient, a suitable representation for the 
probability distributions Pfj,^u{x,ynfi) P^i^uiPi^i^u) should be worked out. For the 
2D square-lattice EA model ([2]), if we represent it by a region-graph shown in figure [2| 
the rgBP equation can be easily implemented. Systematic numerical results on the 2D 
and 3D EA model with bimodal coupling constants will be reported in another paper. 
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